Context-Aware Multimodal Human-Computer Interaction

In the recent years multimodal interfaces have acquired an important role in human computer interaction applications. Humans use multimodality to reduce ambiguity and incompleteness of information. Seemingly they are able to switch easily from one modality to the other and fuse the information from different multimodal sources. The goal of our research was to develop a crisis based human like multimodal system. In particular, we bring into focus the multimodal interaction between human users and the automatic crisis system and its correlation with the adaptability to the human behavior in crisis situations. We have developed a framework of human-computer interaction that offers recognition of various communication modalities including speech, lip movement, facial expression, text and visual symbols. The framework allows the rapid construction of a multimodal, multidevices, and multi-user communication system within crisis management. It enables users to report about the world using different modalities. From the other side the module is able to specify and produce context-sensitive and user-tailored output. By the employment of ontology, it receives the systems view about the world and dialogue actions from an interaction manager and generates appropriate multimodal responses. Siska Fitrianie Man Machine Interaction, Delft University of Technology, Mekelweg 4 2628CD the Netherlands, e-mail: [email protected] Dragos Datcu Man Machine Interaction, Delft University of Technology, Mekelweg 4 2628CD the Netherlands, e-mail: [email protected] Alin G. Chi̧tu Man Machine Interaction, Delft University of Technology, Mekelweg 4 2628CD the Netherlands, e-mail: [email protected] Leon J.M. Rothkrantz Man Machine Interaction, Delft University of Technology, Mekelweg 4 2628CD the Netherlands and Netherlands Defence Academy, Faculty of Technical Sciences, Den Helder, The Netherlands, e-mail: [email protected]